<html>
<head><meta charset="utf-8"><title>Stream the dep-graph #82780 · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html">Stream the dep-graph #82780</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="229113947"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229113947" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229113947">(Mar 06 2021 at 17:21)</a>:</h4>
<p><span class="user-mention" data-user-id="133247">@bjorn3</span> I don't understand your comment:</p>
<blockquote>
<p>I think it would be nice if there was a way for the dep graph to be re-used for a different compilation session inside the same process without serialization. In combination with re-using deserialized metadata, this should make it possible to significantly reduce compilation time for for example a repl I think. This PR would make it impossible to do so. Whether or not this is a significant enough hypothetical future change or not is another question.</p>
</blockquote>
<p>Do you mean in the perspective of a rust interpreter, which has to repeatedly re-traverse the same dep-graph?</p>



<a name="229114252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229114252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229114252">(Mar 06 2021 at 17:26)</a>:</h4>
<p>I am planning to have the dep-graph loading be a simple memmap + some bookkeeping. This would help on the deserialization time.<br>
However, having multiple sessions of the same crate in fly seems odd.</p>



<a name="229114297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229114297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229114297">(Mar 06 2021 at 17:27)</a>:</h4>
<p>Currently when you have a REPL or anything else that could benefit from incremental compilation and that runs within a single process (eg <a href="https://github.com/bjorn3/rustc_codegen_cranelift/issues/1087">bjorn3/rustc_codegen_cranelift#1087</a>), every time a completely new compilation session has to be started that deserializes many things each time. It did be nice if the already deserialized things could be returned from one compilation session and re-used for the next one. This would avoid disk access, deserialization (and optionally serialization) costs. For crate metadata, it should be possible. For the dep graph this would also be possible currently, but after your PR, there is no deserialized dep graph to re-use anymore. Only a serialized dep graph on the disk.</p>



<a name="229114548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229114548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229114548">(Mar 06 2021 at 17:31)</a>:</h4>
<p>In that regard, <a href="https://github.com/rust-lang/rust/issues/82345">#82345</a> may be better suited: only have one dep-graph at all, and clean it at the end of the session.</p>



<a name="229114612"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229114612" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229114612">(Mar 06 2021 at 17:32)</a>:</h4>
<p>Indeed</p>



<a name="229114686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229114686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229114686">(Mar 06 2021 at 17:33)</a>:</h4>
<p>On the other hand that PR has much less of a memory usage reduction. I am kind of split between both approaches.</p>



<a name="229114837"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229114837" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229114837">(Mar 06 2021 at 17:35)</a>:</h4>
<p>Actually, <a href="https://github.com/rust-lang/rust/issues/82780">#82780</a> does not forbid recording the dep-graph, it does it for the <code>query-dep-graph</code> tests. I think your concern is solvable by adding yet one more case to the <code>process_node</code> function.</p>



<a name="229115188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229115188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229115188">(Mar 06 2021 at 17:41)</a>:</h4>
<p>I am fine with landing <a href="https://github.com/rust-lang/rust/issues/82780">#82780</a>. If necessary I will see what I need to do when/if I get around implementing this idea.</p>



<a name="229115980"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Stream%20the%20dep-graph%20%2382780/near/229115980" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Stream.20the.20dep-graph.20.2382780.html#229115980">(Mar 06 2021 at 17:53)</a>:</h4>
<p>Ok, don't hesitate to ping me when you get to it.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>